Conducting automated software testing using centralized controller and distributed test host servers

ABSTRACT

Aspects of the disclosure relate to conducting automated software testing using a centralized controller and one or more distributed test host servers. A computing platform may receive a test execution request. Subsequently, the computing platform may retrieve test specification details information and may identify one or more tests to execute. Then, the computing platform may generate one or more local test execution commands directing a test host server farm to execute the one or more tests. In addition, the one or more local test execution commands may direct the test host server farm to locally construct one or more command line instructions to be executed by the test host server farm to complete the one or more tests. Thereafter, the computing platform may send the one or more remote test execution commands to the test host server farm.

BACKGROUND

Aspects of the disclosure relate to electrical computers, digitalprocessing systems, and automated task management and control. Inparticular, one or more aspects of the disclosure relate to conductingautomated software testing using a centralized controller and one ormore distributed test host servers.

Many organizations have existing investments in automated softwaretesting that often span numerous tools running on multiple platforms(e.g., different operating systems) to test various applications and/orsystems. Typically, the way in which particular tests are launched mayvary based on the tools and platforms being tested. These variances maycreate significant challenges in running, controlling, and/or monitoringtests, as such testing might require subject matter experts who aretrained with the specific platform, the application and/or system, andthe testing tool to be used in conducting a particular test. Further,the results of different tests may have different formats and differentartifacts, which in turn may make it difficult to create unified,standardized, and easily accessible reporting of test results. Moreover,because of the differences across the various platforms, applications,systems, and testing tools, it may be difficult to optimize theefficient and effective technical operations of the hardware andsoftware needed to conduct software testing.

SUMMARY

Aspects of the disclosure provide effective, efficient, scalable, andconvenient technical solutions that address and overcome the technicalproblems associated with testing different software applications and/orcomputing systems across various platforms using various testing toolsby conducting automated software testing using a centralized controllerand one or more distributed test host servers.

In accordance with one or more embodiments, a computing platform havingat least one processor, a memory, and a communication interface mayreceive, via the communication interface, from an administrativecomputing device, a test execution request. Subsequently, the computingplatform may retrieve, from a test specification database, testspecification details information based on the test execution requestreceived from the administrative computing device. Next, the computingplatform may identify one or more tests to execute based on the testspecification details information retrieved from the test specificationdatabase and the test execution request received from the administrativecomputing device. Then, the computing platform may generate one or moreremote test execution commands directing a test host server farm toexecute the one or more tests identified based on the test specificationdetails information retrieved from the test specification database andthe test execution request received from the administrative computingdevice. In addition, generating the one or more remote test executioncommands may include constructing one or more command line instructionsto be executed by the test host server farm and inserting the one ormore command line instructions to be executed by the test host serverfarm into the one or more remote test execution commands. Thereafter,the computing platform may send, via the communication interface, to thetest host server farm, the one or more remote test execution commandsdirecting the test host server farm to execute the one or more testsidentified based on the test specification details information retrievedfrom the test specification database and the test execution requestreceived from the administrative computing device.

In some embodiments, receiving the test execution request from theadministrative computing device may include receiving informationidentifying a specific software application to be tested in a specificcomputing environment.

In some embodiments, after sending the one or more remote test executioncommands to the test host server farm, the computing platform mayremotely capture test output from the test host server farm.Subsequently, the computing platform may store test results data basedon remotely capturing the test output from the test host server farm.

In some embodiments, the one or more command line instructions to beexecuted by the test host server farm may cause the test host serverfarm to run one or more tests that write pass-fail output associatedwith the one or more tests to a console. In some instances, remotelycapturing the test output from the test host server farm may includecapturing the pass-fail output associated with the one or more testsfrom the console.

In some embodiments, remotely capturing the test output from the testhost server farm may include processing the test output captured fromthe test host server farm to produce the test results data. In someinstances, processing the test output captured from the test host serverfarm to produce the test results data may include determining that afirst set of tests passed based on the test output captured from thetest host server farm and determining that a second set of tests failedbased on the test output captured from the test host server farm.

In some embodiments, storing the test results data based on remotelycapturing the test output from the test host server farm may includestoring results information in a test results database and storing testartifacts in a test artifact cloud. In some instances, storing theresults information in the test results database may include storingtimestamps indicating when one or more specific tests started and ended,information identifying a name of a specific host on which the one ormore specific tests were run, information identifying a number ofassertions that passed, and information identifying a number ofassertions that failed.

In some embodiments, prior to receiving the test execution request fromthe administrative computing device, the computing platform may receiveat least a portion of the test specification details information fromenterprise server architecture. Subsequently, the computing platform maystore the at least the portion of the test specification detailsinformation in the test specification database.

In some embodiments, the computing platform may receive, via thecommunication interface, from a client computing device, a request fortest results information. In response to receiving the request for thetest results information from the client computing device, the computingplatform may generate a test results report. Subsequently, the computingplatform may send, via the communication interface, to the clientcomputing device, the test results report.

In some instances, receiving the request for the test resultsinformation from the client computing device may include receiving therequest for the test results information via a web dashboard provided tothe client computing device. In some instances, receiving the requestfor the test results information from the client computing device mayinclude receiving the request for the test results information via anapplication programming interface.

In accordance with one or more additional embodiments, a computingplatform having at least one processor, a memory, and a communicationinterface may receive, via the communication interface, from anadministrative computing device, a test execution request. Subsequently,the computing platform may retrieve, from a test specification database,test specification details information based on the test executionrequest received from the administrative computing device. Next, thecomputing platform may identify one or more tests to execute based onthe test specification details information retrieved from the testspecification database and the test execution request received from theadministrative computing device.

Then, the computing platform may generate one or more local testexecution commands directing a test host server farm to execute the oneor more tests identified based on the test specification detailsinformation retrieved from the test specification database and the testexecution request received from the administrative computing device. Inaddition, the one or more local test execution commands may direct thetest host server farm to locally construct one or more command lineinstructions to be executed by the test host server farm to complete theone or more tests. Thereafter, the computing platform may send, via thecommunication interface, to the test host server farm, the one or morelocal test execution commands directing the test host server farm toexecute the one or more tests identified based on the test specificationdetails information retrieved from the test specification database andthe test execution request received from the administrative computingdevice.

In some embodiments, receiving the test execution request from theadministrative computing device may include receiving informationidentifying a specific software application to be tested in a specificcomputing environment.

In some embodiments, after sending the one or more local test executioncommands to the test host server farm, the computing platform mayreceive, via a tests results service, test results data from the testhost server farm. Subsequently, the computing platform may store thetest results data received from the test host server farm via the testsresults service in a test results database.

In some embodiments, the one or more local test execution commands maycause a client application executed on the test host server farm to runone or more tests on the test host server farm that write pass-failoutput associated with the one or more tests to a console on the testhost server farm.

In some embodiments, the one or more local test execution commands maycause the client application executed on the test host server farm tolocally capture output from the one or more tests run on the test hostserver farm. In some instances, the one or more local test executioncommands may cause the client application executed on the test hostserver farm to locally process the output captured from the one or moretests run on the test host server farm.

In some embodiments, the one or more local test execution commands maycause the client application executed on the test host server farm tolocally process the output captured from the one or more tests run onthe test host server farm by determining, at the test host server farm,that a first set of tests passed based on the output captured from theone or more tests run on the test host server farm and by determining,at the test host server farm, that a second set of tests failed based onthe output captured from the one or more tests run on the test hostserver farm.

In some embodiments, storing the test results data received from thetest host server farm via the tests results service in the test resultsdatabase may include storing timestamps indicating when one or morespecific tests started and ended, information identifying a name of aspecific host on which the one or more specific tests were run,information identifying a number of assertions that passed, andinformation identifying a number of assertions that failed.

In some embodiments, after sending the one or more local test executioncommands to the test host server farm, the computing platform mayreceive, via the tests results service, test output data from the testhost server farm. Subsequently, the computing platform may store thetest output data received from the test host server farm via the testsresults service in a test artifact cloud.

In some embodiments, prior to receiving the test execution request fromthe administrative computing device, the computing platform may receiveat least a portion of the test specification details information fromenterprise server architecture. Subsequently, the computing platform maystore the at least the portion of the test specification detailsinformation in the test specification database.

In some embodiments, the computing platform may receive, via thecommunication interface, from a client computing device, a request fortest results information. In response to receiving the request for thetest results information from the client computing device, the computingplatform may generate a test results report. Subsequently, the computingplatform may send, via the communication interface, to the clientcomputing device, the test results report.

In some instances, receiving the request for the test resultsinformation from the client computing device may include receiving therequest for the test results information via a web dashboard provided tothe client computing device. In some instances, receiving the requestfor the test results information from the client computing device mayinclude receiving the request for the test results information via anapplication programming interface.

These features, along with many others, are discussed in greater detailbelow.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example and not limitedin the accompanying figures in which like reference numerals indicatesimilar elements and in which:

FIGS. 1A and 1B depict an illustrative computing environment forconducting automated software testing using a centralized controller andone or more distributed test host servers in accordance with one or moreexample embodiments;

FIGS. 1C and 1D depict additional illustrative computing environmentsfor conducting automated software testing using a centralized controllerand one or more distributed test host servers in accordance with one ormore example embodiments;

FIGS. 2A, 2B, and 2C depict an illustrative event sequence forconducting automated software testing using a centralized controller andone or more distributed test host servers in accordance with one or moreexample embodiments;

FIGS. 3A, 3B, 3C, 3D, and 3E depict another illustrative event sequencefor conducting automated software testing using a centralized controllerand one or more distributed test host servers in accordance with one ormore example embodiments;

FIGS. 4, 5, 6, and 7 depict example graphical user interfaces forconducting automated software testing using a centralized controller andone or more distributed test host servers in accordance with one or moreexample embodiments;

FIG. 8 depicts an illustrative method for conducting automated softwaretesting using a centralized controller and one or more distributed testhost servers in accordance with one or more example embodiments; and

FIG. 9 depicts another illustrative method for conducting automatedsoftware testing using a centralized controller and one or moredistributed test host servers in accordance with one or more exampleembodiments.

DETAILED DESCRIPTION

In the following description of various illustrative embodiments,reference is made to the accompanying drawings, which form a parthereof, and in which is shown, by way of illustration, variousembodiments in which aspects of the disclosure may be practiced. It isto be understood that other embodiments may be utilized, and structuraland functional modifications may be made, without departing from thescope of the present disclosure.

It is noted that various connections between elements are discussed inthe following description. It is noted that these connections aregeneral and, unless specified otherwise, may be direct or indirect,wired or wireless, and that the specification is not intended to belimiting in this respect.

FIGS. 1A and 1B depict an illustrative computing environment forconducting automated software testing using a centralized controller andone or more distributed test host servers in accordance with one or moreexample embodiments. Referring to FIG. 1A, computing environment 100 mayinclude one or more computing platforms, one or more computing devices,and various other computing infrastructure. For example, computingenvironment 100 may include software testing computing platform 110,administrative computing device 120, client computing device 130,enterprise server architecture 140, test host server farm 150, test hostserver farm 160, and test host server farm 170.

As illustrated in greater detail below, software testing computingplatform 110 may include one or more computing devices configured toperform one or more of the functions described herein. For example,software testing computing platform 110 may include one or morecomputers (e.g., laptop computers, desktop computers, servers, serverblades, or the like) configured to perform one or more of the functionsdescribed herein, as discussed in greater detail below.

Administrative computing device 120 may be configured to provide one ormore interfaces that allow for configuration and management of one ormore other computing devices and/or computer systems included incomputing environment 100, such as one or more interfaces that allow forconfiguration and management of software testing computing platform 110.Client computing device 130 may be configured to provide one or moreinterfaces that allow for interaction with and use of one or more othercomputing devices and/or computer systems included in computingenvironment 100, such as one or more interfaces that allow forinteraction with and use of software testing computing platform 110.

Enterprise server architecture 140 may include multiple server computersthat are owned, operated, maintained, and/or otherwise used by anorganization, such as an enterprise organization associated withsoftware testing computing platform 110. For example, enterprise serverarchitecture 140 may include one or more server computers that storeand/or otherwise maintain enterprise applications (which may, e.g., beexecuted by and/or provided to one or more computing devices associatedwith enterprise users) and/or enterprise information (which may, e.g.,be accessed and/or used by the enterprise applications and/or by the oneor more computing devices associated the with enterprise users).

For example, enterprise server architecture 140 may include one or morecomputer systems that are configured to provide one or more portalinterfaces to one or more client devices and/or configured toauthenticate one or more client devices and/or users of such devices tosuch portal interfaces. For example, enterprise server architecture 140may include a computer system configured to provide a customer portal,such as an online banking portal, to one or more customers of anorganization, such as a financial institution, who may use one or morecomputing devices to access the portal and/or who may be authenticatedto one or more portal user accounts by the computer system using variousauthentication techniques. In some instances, in addition to beingconfigured to provide an online banking portal associated with afinancial institution to one or more customers of the financialinstitution and/or their associated computing devices, the computersystem (which may, e.g., be included in enterprise server architecture140) also may be configured to provide a mobile banking portalassociated with the financial institution to various customers of thefinancial institution and/or their associated mobile computing devices.Such portals may, for instance, provide customers of the financialinstitution with access to financial account information (e.g., accountbalance information, account statements, recent transaction historyinformation, or the like) and/or may provide customers of the financialinstitution with menus, controls, and/or other options to scheduleand/or execute various transactions (e.g., online bill pay transactions,person-to-person funds transfer transactions, or the like).

Additionally or alternatively, enterprise server architecture 140 mayinclude one or more client account servers, which may be configured tostore and/or maintain information associated with one or more clientaccounts. For example, the client account server(s) may be configured tostore and/or maintain information associated with one or more financialaccounts associated with one or more customers of a financialinstitution, such as account balance information, transaction historyinformation, and/or the like. Additionally or alternatively, the clientaccount server(s) may include and/or integrate with one or more clientsupport servers and/or devices, such as one or more customer servicerepresentative devices used by one or more customer servicerepresentatives of an organization (which may, e.g., be a financialinstitution operating one or more computer systems in computingenvironment 100), to connect one or more customers of the organizationwith one or more customer service representatives of the organizationvia one or more telephone support sessions, chat support sessions,and/or other client support sessions.

Test host server farm 150, test host server farm 160, and test hostserver farm 170 may be distinct and physically separate data centers orother groupings of server computers that are operated by and/orotherwise associated with an organization, such as a financialinstitution. In addition, each of test host server farm 150, test hostserver farm 160, and test host server farm 170 may house a plurality ofserver computers and various other computers, network components, anddevices. For example, each of test host server farm 150, test hostserver farm 160, and test host server farm 170 may include a pluralityof server nodes that are made up of and/or otherwise include one or moreservers and/or server blades, which may be monitored and/or controlledby software testing computing platform 110 and/or one or more othercomputing devices included in computing environment 100. Each serverand/or server blade included in the plurality of server nodes associatedwith test host server farm 150, test host server farm 160, and test hostserver farm 170 may include one or more processors, memories,communication interfaces, storage devices, and/or other components.

Computing environment 100 also may include one or more networks, whichmay interconnect one or more of software testing computing platform 110,administrative computing device 120, client computing device 130,enterprise server architecture 140, test host server farm 150, test hostserver farm 160, and test host server farm 170. For example, computingenvironment 100 may include network 180, which may include one or morepublic networks, one or more private networks, and/or one or moresub-networks (e.g., local area networks (LANs), wide area networks(WANs), or the like).

In one or more arrangements, administrative computing device 120, clientcomputing device 130, and other computing devices included in computingenvironment 100 may be any type of computing device capable of receivinga user interface, receiving input via the user interface, andcommunicating the received input to one or more other computing devices.For example, administrative computing device 120 and client computingdevice 130 may, in some instances, be and/or include server computers,desktop computers, laptop computers, tablet computers, smart phones, orthe like and may include one or more processors, memories, communicationinterfaces, storage devices, and/or other components. As noted above,and as illustrated in greater detail below, any and/or all of thecomputing devices included in computing environment 100 may, in someinstances, be special-purpose computing devices configured to performspecific functions.

Referring to FIG. 1B, software testing computing platform 110 mayinclude one or more processors 111, one or more memories 112, and one ormore communication interfaces 113. A data bus may interconnect the oneor more processors 111, the one or more memories 112, and the one ormore communication interfaces 113. Communication interface 113 may be anetwork interface configured to support communication between softwaretesting computing platform 110 and one or more networks (e.g., network180 or the like). Memory 112 may include one or more program moduleshaving instructions that when executed by processor 111 cause softwaretesting computing platform 110 to perform one or more functionsdescribed herein and/or one or more databases and/or other librariesthat may store and/or otherwise maintain information which may be usedby such program modules and/or processor 111.

In some instances, the one or more program modules and/or databases maybe stored by and/or maintained in different memory units of softwaretesting computing platform 110 and/or by different computing devicesthat may form and/or otherwise make up software testing computingplatform 110. For example, memory 112 may have, store, and/or include asoftware testing service module 112 a, a test specification database 112b, a test results service module 112 c, a test report service module 112d, a test results database 112 e, and a test artifact cloud 112 f.Software testing service module 112 a may include hardware componentsand/or executable instructions that enable and/or cause software testingcomputing platform 110 to provide a software testing service, such asthe test master service discussed below. Test specification database 112b may store test specification details and/or other informationassociated with conducting software application testing. Test resultsservice module 112 c may include hardware components and/or executableinstructions that enable and/or cause software testing computingplatform 110 to provide a test results service, such as the test resultsservice discussed below. Test report service module 112 d may includehardware components and/or executable instructions that enable and/orcause software testing computing platform 110 to provide a testreporting service, such as the test report service discussed below. Testresults database 112 e may store test results data and/or otherinformation that may result from and/or be determined based onconducting software application testing. Test artifact cloud 112 f maystore test output data and/or other information that may be capturedduring software application testing.

FIGS. 1C and 1D depict additional illustrative computing environmentsfor conducting automated software testing using a centralized controllerand one or more distributed test host servers in accordance with one ormore example embodiments. In one or more arrangements, software testingcomputing platform 110 and/or the other elements of computingenvironment 100 discussed above in connection with FIGS. 1A and 1B maybe configured in the manner illustrated in FIGS. 1C and 1D and/orotherwise be arranged to provide the architectures illustrated in FIGS.1C and 1D and discussed in greater detail below.

In one or more arrangements, the architectures illustrated in FIGS. 1Cand 1D may implement and/or provide a platform-agnostic andtechnology-agnostic test execution harness. In particular, these examplearchitectures may provide a clean interface (which may, e.g., beprovided as a user interface (UI) and/or as an application programminginterface (API)) for scheduling tests, maintaining a centralized resultsrepository to store test results and/or provide reporting servicesassociated with such tests. In addition, various aspects may beimplemented without requiring developer and/or tester teams to changethe one or more testing tool(s) that they may be currently using.

In one or more arrangements, the architectures illustrated in FIGS. 1Cand 1D may be used to launch a test remotely on a target host, collectand analyze the output to determine pass and/or fail assertion counts,and store the results. As discussed below, this may be achieved withmultiple levels of complexity depending on different features. Someaspects of the disclosure facilitate the execution of tests that meetthree criteria: (1) the test can be launched from a command line/systemcall; (2) the test can write information to standard-out (“stdout”); and(3) the test owner can provide two simple regular expressions (“regex”)or XPath expressions (“XPath”) for use by the test architectures inanalyzing the output generated in testing to identify passing andfailing assertions. With respect to the first criterion, many testingtools can be launched from a command line/system call. The exact commandline or system call may vary depending on the operating system of thehost where test execution occurs. With respect to the second criterion,many testing tools can write information to standard-out or thisfunctionality can be simulated. Almost all testing tools generate someform of output, like a log of all of the interactions with theapplication under test (“AUT”) and messages regarding the pass or failstatus of each assertion. If that information is not already written tostdout by a particular testing tool, adding a single “print( )”statement in the core logic of the test may cause the tool to write tostdout. Alternatively, some tools may capture those messages local file(e.g., in XML or in another format) which can be written to stdout bythe testing tool at the end of test execution by using ‘cat <filename>’(Unix) or ‘type <filename>’ (Windows) or similar. With respect to thethird criterion, many testing tools provide structured test output. Forexample, for a test where the lines of output begin with the string“Pass:” or “Fail:” (followed by some details of the scenario thatpassed) the regex “^Pass:*” and “^Fail:*” would be correct. Given theuse of XPath in test output and the flexibility of regex, most allstructured output can be matched to an appropriate set of XPathexpressions or pair of pass/fail regex strings.

In one or more arrangements, the architectures illustrated in FIGS. 1Cand 1D may involve deploying one or more components. For instance, thearchitecture may utilize enterprise components, such as a host lookupfacility (e.g., a service or database) that identifies the operatingsystem running on any testing host. In addition, the architecture mayutilize test hosts and/or farms where tests may be executed. Thearchitecture also may utilize a “TestAll Spec Database” where developersand/or testers may provide details of each test to be run by the testingservice. These details may include a unique logical name of the test,unique application name(s)/ID(s) of application(s) tested by theautomated test, command line details to launch the test, pass/fail regexstrings, and a default host/class where the test should execute. Thearchitecture also may utilize a “TestAll Artifact Cloud” key/valueobject storage platform to serve as the repository of all test executionartifacts, including test output, screen shots, and/or the like.

In some arrangements, the architecture also may utilize a “TestAllResult Database” to store test execution details: test name, unique testrun ID, application name(s)/ID(s), timestamps (test scheduled, teststarted, test completed), hostname where execution occurred, key(s) toobtain test execution artifacts, assertion counts (passed, failed) andidentifier of the person and/or application that submitted the testexecution request to the testing service. The architecture also mayutilize a “TestAll Master Service” that accepts test execution requestsvia API. The API signature may include the unique name of the test to berun, a start-after time (e.g., for use when scheduling tests earlierthan desired execution time) and testing host/class (e.g., for use whenoverriding the default listed in the “Test Spec” database). The “TestAllMaster” may also include an API for providing programmatic updates tothe “Test Spec” data.

In some arrangements, the architecture also may include a “TestAllReport Service” that accepts requests for test results data via API. TheAPI signature may include search parameters including test name, uniquetest run ID, application name(s)/ID(s) and timestamps as searchcriteria. The service may return the matching rows from the “TestResult” database. In some instances, this service also may generateon-demand document-formatted reports and/or the like. The architecturealso may include a “TestAll Control Panel” that provides a userinterface for developers and/or testers to create or update “Test Spec”details and to submit on-demand test execution requests. Thearchitecture also may include a “TestAll Report Panel” that provides auser interface for developers and/or testers to view test results basedon flexible search criteria (which may, e.g., be specified in the“TestAll Control Panel”). The results may include links to viewableartifacts stored in the “TestAll Artifact” repository (e.g., testoutput, screen shots, and/or the like).

In some arrangements, the architecture also may include a “TestAllResults Service” that provides a clean API for TestAll Clients to submittest results and artifacts for storage in the TestAll Results Databaseand TestAll Artifact Cloud. In addition, the architecture may include a“TestAll Client” that provides a daemon process which may run on eachtest host/farm and performs test execution, post-processing, and resultssubmission tasks.

Referring to FIG. 1C, a “basic architecture” is illustrated and arelated process flow is described below. In a Test Setup phase, usersmay submit test specifications to the TestAll Master Service via an APIthat can be accessed from the TestAll Control Panel or from other tools.In a Test Execution Request Submission phase, a request for testexecution may be submitted to the TestAll Master Service via an API thatcan be accessed from the TestAll Control Panel. In some instances, testsexecution requests may be submitted via the API from other systems. Forexample, upon successful distribution of a new application build to atesting environment, a build system may call the API to automaticallylaunch the associated tests. In another example, a development ortesting team wanting to re-run tests may do so from their own qualityassurance dashboard which would call the TestAll Master Service API. Inanother example, for automated daily unattended test reruns, a scheduledscript (cron, autosys, and/or the like) may call the TestAll MasterService API.

In a Test Execution phase, after receiving a test execution request, theTestAll Master Service may pull information about the test from theTestAll Spec DB, check the operating system of the target host from thehost lookup facility and then generate the correct command to launch thetest remotely on the target testing host. The command may be craftedsuch that the standard output is redirected or piped to a local file onthe TestAll Master Service server. Upon completion of the testexecution, the TestAll Master Server may pass the output through apost-parser that uses the regular expressions from the test's entry inthe TestAll Spec DB to determine the number of passed and failedassertions in the test.

In a Test Results and Artifacts Storage phase, after post-processing iscomplete, the TestAll Master Server may post the test output in theTestAll Artifact Cloud and store test results in the TestAll ResultsDatabase. In a Test Results Reporting phase, test results may be queriedusing the TestAll Report Service API using various tools (e.g., testresults dashboards) or using the TestAll Report Panel.

The architecture presented in FIG. 1C presents a number of technicaladvantages. For instance, this approach may be easy to implement. Inaddition, the test execution hosts might not require TestAll software tobe installed locally. Some difficulties that may be presented by thisapproach, however, include the load on the TestAll Master Service. Inaddition, if different tests need to run under different usernames(e.g., service IDs), it may be necessary to maintain information in theTestAll Spec Database regarding the correct username (service ID) foreach test to run under and the test owners might have to ensure that theappropriate access credentials are established and managed for TestAllto be able to run processes on those hosts. In some instances, it alsomay be challenging for the TestAll Master Service to manage the numberof concurrent tests running on each host. This may lead to hostsbecoming overwhelmed by concurrent execution requests and multipleWindows-based UI tests from running concurrently and causing clash. Inaddition, the test artifact that flows into the TestAll Artifact Cloudmight be limited to the test output in some instances.

Referring to FIG. 1D, an “advanced architecture” is illustrated and arelated process flow is described below. In the advanced architecture,each test execution host may run a daemon (“TestAll Client”) processwhich manages local test execution, post-processing and resultssubmission. The Test Setup phase and the Test Execution RequestSubmission phase may proceed in the same manner as discussed above withrespect to the basic architecture.

In a Test Execution phase, after receiving a test execution request, theTestAll Master Service may send a message to the TestAll Client on thetarget host/farm where that test needs to run. The message may containhost command, execution username/service ID and pass/fail regex detailsfrom the TestAll Spec DB. The TestAll Client may manage a queue of testsrequiring execution and may only run the number of tests appropriate forthat host (e.g., based on criteria such as available resources or amaximum threshold defined by the host owner). The TestAll Client maycheck the operating system on the local host (e.g., no need to query ahost lookup service) and may generate the correct command to launch thetest locally. The command may be crafted such that the standard outputis redirected or piped to a local file on the test host. If the requiredusername/service ID does not match the username/service ID that isrunning the TestAll Client itself, then the TestAll Client may craft thecommand accordingly to allow the test to run under the requiredexecution username/service ID. Upon completion of the test execution,the TestAll Client may pass the output through a post-parser that usesthe regular expressions to determine the number of passed and failedassertions in the test.

In a Test Results and Artifacts Storage phase, after post-processing iscomplete, the TestAll Client may post the test output in the TestAllArtifact Cloud and store test results in the TestAll Results Database.This storage may, in some instances, be performed via a resultssubmission service, rather than direct database access. The Test ResultsReporting phase may proceed in the same manner as discussed above withrespect to the basic architecture.

The architecture presented in FIG. 1D presents a number of technicaladvantages. For instance, in this approach, the TestAll client may besetup to launch upon system boot under the correct username/service IDfor tests run on that host. Thus, if the host does not need to run testsrequiring multiple usernames/service IDs, no further credentialmanagement might be needed. In addition, managing the number ofconcurrent tests run on any given host may be managed by the TestAllClient, which may be advantageous. In some instances, however, it may bemore challenging to implement and the test execution hosts may requireTestAll software to be installed locally. In some instances, it may benecessary to manage the test execution queues such that if a TestAllClient is rebooted, the test execution queue on that host can berecovered. Similarly, there may be a need for a facility to cancel teststhat were already scheduled and reside in the TestAll Client's queue.

In some instances, it may be important that the version of theapplication under test (“AUT”) is carefully considered. For example, iftests are intended to run on a particular application version, the testsshould not be allowed to launch if that version is missing on the testhost. One example: The current test execution hosts have version 2.01 ofan application. An automation platform builds version 2.02 of HelloWorldand initiates deployment of that version to the testing environment, andthen immediately calls the TestAll API to run the tests of HelloWorld.This may create a race condition in which TestAll could begin to launchthe tests before version 2.02 is available on the hosts. (The delay ofversion 2.02 may occur in a software deployment fabric, for example.) Itmay be desired for the tests to not run until version 2.02 is present.This may be solved in the following way: If the test requestor (a personor a system which calls the TestAll Master Service execution API) wantsto ensure that the test is only run on a particular version of thesoftware, they may specify that in the API call. The installed versionof HelloWorld may be verified before launching the tests, and if thewrong version is present then the test would not run. Implementing thismay be done by checking software versions (or for the mechanism to checka particular software's version to be part of the TestAll Spec DB datasupplied in the test specification details). In some instances, thismechanism may be implemented in the Advanced Architecture using theTestAll Client.

In some instances, in both the Basic Architecture and AdvancedArchitecture, it may be necessary to enforce a maximum time limit fortext execution and to kill “zombie” tests (any tests that overrun themaximum time). The maximum time may have a default amount, and may beoverridden by providing a value in the test specification details storedin the TestAll Spec DB. Enforcing the maximum time may be achieved byusing a timeout wrapper command that kills the child process in whichthe test runs. In the Advanced Architecture model, this may be doneusing the TestAll Client to enforce the maximum execution time.

In some instances, when running large numbers of automated tests acrosslarge numbers of testing hosts, specifying exact hostnames in the testspecification details stored in the TestAll Spec DB might not bepreferred. Rather, it may be better to have clusters of hosts which aremembers of a test farm, and a test may be executed on any member of theappropriate farm. TestAll may leverage existing cloud/FaaSinfrastructure to handle either specific hostname or farm name in thetest specification details.

In some instances, unattended execution of Windows-based user interfacetests may be challenging because the host where the test is executedmust have the screen, keyboard and mouse unlocked (no screen saver orlock screen). This issue might need to be addressed and resolved beforeTestAll may launch unattended UI tests on Windows hosts. In someinstances, this issue may be resolved by establishing a protected classof hosts with the lock security policy disabled in a controlledenvironment acceptable to the organization and/or its informationsecurity group.

FIGS. 2A, 2B, and 2C depict an illustrative event sequence forconducting automated software testing using a centralized controller andone or more distributed test host servers in accordance with one or moreexample embodiments. The example event sequence depicted in FIGS. 2A,2B, and 2C may, for instance, illustrate a series of steps executed inconnection with the basic architecture discussed above.

Referring to FIG. 2A, at step 201, software testing computing platform110 may receive test specification details information (e.g., fromenterprise server architecture 140 and/or from one or more othersources). For example, at step 201, prior to receiving a test executionrequest, software testing computing platform 110 may receive at least aportion of test specification details information from enterprise serverarchitecture (e.g., enterprise server architecture 140). The testspecification details information received by software testing computingplatform 110 from enterprise server architecture 140 may, for instance,include: information specifying one or more specific, different tests torun for one or more specific, different types of applications;information specifying one or more specific hosts and/or specific typesof hosts those tests should be run on; information specifying how tolaunch those tests (e.g., information specifying command line functions,arguments, parameters, and/or other launch information); and/or thelike. The tests that may be defined in the test specification detailsinformation received by software testing computing platform 110 fromenterprise server architecture 140 and/or subsequently run by softwaretesting computing platform 110 and/or associated testing architecturemay include any type of software application tests, such assecurity-related tests, server-side tests and/or service-basedarchitecture tests, fat-client and/or heavy-user-interface tests,user-application tests, web-based application tests, and/or other typesof tests.

At step 202, software testing computing platform 110 may update a testspecification database based on the test specification detailsinformation received from enterprise server architecture 140. Forexample, at step 202, software testing computing platform 110 may storethe at least the portion of the test specification details information(which may, e.g., be received from enterprise server architecture 140)in a test specification database (e.g., test specification database 112b).

At step 203, software testing computing platform 110 may receive a testexecution request (e.g., from administrative computing device 120,identifying a particular application to be tested and a particularenvironment for conducting such testing). For example, at step 203,software testing computing platform 110 may receive, via a communicationinterface (e.g., communication interface 113), from an administrativecomputing device (e.g., administrative computing device 120), a testexecution request. In some embodiments, receiving the test executionrequest from the administrative computing device may include receivinginformation identifying a specific software application to be tested ina specific computing environment. For example, in receiving the testexecution request from the administrative computing device (e.g.,administrative computing device 120), software testing computingplatform 110 may receive information identifying a specific softwareapplication to be tested in a specific computing environment.

In some instances, in receiving the test execution request fromadministrative computing device 120, software testing computing platform110 may receive a message and/or other information from administrativecomputing device 120 corresponding to one or more selections made viaone or more graphical user interfaces presented by administrativecomputing device 120, such as graphical user interface 400, which isdepicted in FIG. 4. As seen in FIG. 4, graphical user interface 400 mayinclude one or more user interface elements enabling a user ofadministrative computing device 120 to specify a particular applicationto be tested and a particular environment for testing the application,specify other options associated with the testing, and/or initiate thetest execution request.

At step 204, software testing computing platform 110 may load testspecification details information from a test specification databasebased on the test execution request. For example, at step 204, softwaretesting computing platform 110 may retrieve, from a test specificationdatabase (e.g., test specification database 112 b), test specificationdetails information based on the test execution request received fromthe administrative computing device (e.g., administrative computingdevice 120).

Referring to FIG. 2B, at step 205, software testing computing platform110 may identify one or more tests to execute based on the testspecification details information and the test execution request. Forexample, at step 205, software testing computing platform 110 mayidentify one or more tests to execute based on the test specificationdetails information retrieved from the test specification database(e.g., test specification database 112 b) and the test execution requestreceived from the administrative computing device (e.g., administrativecomputing device 120). Software testing computing platform 110 may, forinstance, identify the one or more tests to execute by matchinginformation from the test execution request (which may, e.g., specify aparticular application to be tested in a particular environment) withcorresponding test specification details information (which may, e.g.,specify what tests should be run to facilitate testing of the particularapplication, where such tests should be run, how such tests should belaunched, and/or the like) so as to initiate and/or complete testing ofthe specified application in the specified environment.

At step 206, software testing computing platform 110 may generate one ormore remote execution commands directing test host server farm 150 toexecute the one or more identified tests. For example, at step 206,software testing computing platform 110 may generate one or more remotetest execution commands directing a test host server farm (e.g., testhost server farm 150) to execute the one or more tests identified basedon the test specification details information retrieved from the testspecification database (e.g., test specification database 112 b) and thetest execution request received from the administrative computing device(e.g., administrative computing device 120). In addition, generating theone or more remote test execution commands may include constructing oneor more command line instructions to be executed by the test host serverfarm and inserting the one or more command line instructions to beexecuted by the test host server farm into the one or more remote testexecution commands. For example, in generating the one or more remotetest execution commands (e.g., at step 206), software testing computingplatform 110 may construct one or more command line instructions to beexecuted by the test host server farm (e.g., test host server farm 150).Then, software testing computing platform 110 may insert the one or morecommand line instructions to be executed by the test host server farm(e.g., test host server farm 150) into the one or more remote testexecution commands. In this way, software testing computing platform 110may remotely execute various software application tests on test hostserver farm 150 (e.g., by constructing the command line, launchingvarious tests on a particular application, and then capturing andprocessing the output of such tests).

At step 207, software testing computing platform 110 may send the one ormore remote execution commands to test host server farm 150. Forexample, at step 207, software testing computing platform 110 may send,via the communication interface (e.g., communication interface 113), tothe test host server farm (e.g., test host server farm 150), the one ormore remote test execution commands directing the test host server farm(e.g., test host server farm 150) to execute the one or more testsidentified based on the test specification details information retrievedfrom the test specification database (e.g., test specification database112 b) and the test execution request received from the administrativecomputing device (e.g., administrative computing device 120).

At step 208, software testing computing platform 110 may remotelycapture test output from test host server farm 150. For example, at step208, after sending the one or more remote test execution commands to thetest host server farm (e.g., test host server farm 150), softwaretesting computing platform 110 may remotely capture test output from thetest host server farm (e.g., test host server farm 150). In remotelycapturing test output from test host server farm 150, software testingcomputing platform 110 may capture raw test output and/or may processthe raw test output. For instance, software testing computing platform110 may harvest and/or otherwise capture what is occurring on test hostserver farm 150 as the one or more tests are being executed and mayinterrogate data received from test host server farm 150 to determinewhether specific tests are passing or failing. In addition, softwaretesting computing platform 110 may write results data to a resultsdatabase (e.g., test results database 112 e).

In some instances, while remotely capturing test output from test hostserver farm 150, software testing computing platform 110 may generateone or more user interfaces having status information indicative of theprogress of the testing. In addition, software testing computingplatform 110 may send such user interfaces to administrative computingdevice 120 and/or otherwise cause administrative computing device 120 todisplay such user interfaces. For example, while remotely capturing testoutput from test host server farm 150, software testing computingplatform 110 may generate and send to administrative computing device120 a graphical user interface similar to graphical user interface 500,which is depicted in FIG. 5. As seen in FIG. 5, graphical user interface500 may include status information identifying the application currentlybeing tested and the environment in which the application is beingtested, progress information identifying how far along the testing is,and/or other user interface elements enabling a user of administrativecomputing device 120 to specify other options associated with thetesting.

In some embodiments, the one or more command line instructions to beexecuted by the test host server farm may cause the test host serverfarm to run one or more tests that write pass-fail output associatedwith the one or more tests to a console. For example, the one or morecommand line instructions to be executed by the test host server farm(which may, e.g., be generated and sent by software testing computingplatform 110 to test host server farm 150) may cause the test hostserver farm (e.g., test host server farm 150) to run one or more teststhat write pass-fail output associated with the one or more tests to aconsole. For instance, the one or more command line instructions to beexecuted by the test host server farm (which may, e.g., be generated andsent by software testing computing platform 110 to test host server farm150) may cause the test host server farm (e.g., test host server farm150) to run one or more tests that write pass-fail output associatedwith the one or more tests to stdout, as discussed in the examplesabove.

In some embodiments, remotely capturing the test output from the testhost server farm may include capturing the pass-fail output associatedwith the one or more tests from the console. For example, in remotelycapturing the test output from the test host server farm (e.g., testhost server farm 150), software testing computing platform 110 maycapture the pass-fail output associated with the one or more tests fromthe console. For instance, software testing computing platform 110 maycapture the pass-fail output associated with the one or more tests fromthe console on test host server farm 150 using one or more regularexpressions and/or other output capture tools, as discussed in theexamples above.

In some embodiments, remotely capturing the test output from the testhost server farm may include processing the test output captured fromthe test host server farm to produce test results data. For example, inremotely capturing the test output from the test host server farm (e.g.,test host server farm 150), software testing computing platform 110 mayprocess the test output captured from the test host server farm (e.g.,test host server farm 150) to produce test results data. The testresults data may, for instance, include information identifying specifictests that passed, specific tests that failed, specific hosts thatexecuted specific tests, and/or other information associated with thetesting.

In some embodiments, processing the test output captured from the testhost server farm to produce the test results data may includedetermining that a first set of tests passed based on the test outputcaptured from the test host server farm and determining that a secondset of tests failed based on the test output captured from the test hostserver farm. For example, in processing the test output captured fromthe test host server farm (e.g., test host server farm 150) to producethe test results data, software testing computing platform 110 maydetermine that a first set of tests passed based on the test outputcaptured from the test host server farm (e.g., test host server farm150) and may determine that a second set of tests failed based on thetest output captured from the test host server farm (e.g., test hostserver farm 150).

Referring to FIG. 2C, at step 209, software testing computing platform110 may store test results data. For example, at step 209, softwaretesting computing platform 110 may store test results data based onremotely capturing the test output from the test host server farm (e.g.,test host server farm 150). Software testing computing platform 110 may,for instance, store test results in test results database 112 e and maycapture and/or store artifacts in test artifact cloud 112 f. The testresults also may, for instance, include timestamps indicating whenparticular tests started and/or ended, the names of the hosts on whichparticular tests ran, how many assertions passed and/or failed, and/orother information associated with the testing.

In some embodiments, storing the test results data based on remotelycapturing the test output from the test host server farm may includestoring results information in a test results database and storing testartifacts in a test artifact cloud. For example, in storing the testresults data based on remotely capturing the test output from the testhost server farm (e.g., test host server farm 150), software testingcomputing platform 110 may store results information in a test resultsdatabase (e.g., test results database 112 e) and may store testartifacts in a test artifact cloud (e.g., test artifact cloud 112 f).

In some embodiments, storing the results information in the test resultsdatabase may include storing timestamps indicating when one or morespecific tests started and ended, information identifying a name of aspecific host on which the one or more specific tests were run,information identifying a number of assertions that passed, andinformation identifying a number of assertions that failed. For example,in storing the results information in the test results database (e.g.,test results database 112 e), software testing computing platform 110may store timestamps indicating when one or more specific tests startedand ended on test host server farm 150, information identifying a nameof a specific host associated with test host server farm 150 on whichthe one or more specific tests were run, information identifying anumber of assertions that passed during the testing executed on testhost server farm 150, information identifying a number of assertionsthat failed during the testing executed on test host server farm 150,and/or other information associated with the testing executed on testhost server farm 150.

At step 210, software testing computing platform 110 may receive areport request (e.g., from client computing device 130). For example, atstep 210, software testing computing platform 110 may receive, via thecommunication interface (e.g., communication interface 113), from aclient computing device (e.g., client computing device 130), a requestfor test results information. In some instances, in receiving the reportrequest from client computing device 130, software testing computingplatform 110 may receive a message and/or other information from clientcomputing device 130 corresponding to one or more selections made viaone or more graphical user interfaces presented by client computingdevice 130, such as graphical user interface 600, which is depicted inFIG. 6, and/or graphical user interface 700, which is depicted in FIG.7. As seen in FIG. 6, graphical user interface 600 may include one ormore user interface elements enabling a user of client computing device130 to access a test summary report associated with the applicationtesting controlled by software testing computing platform 110, accessdetailed test results associated with the application testing controlledby software testing computing platform 110, and/or access other optionsassociated with the application testing controlled by software testingcomputing platform 110. As seen in FIG. 7, graphical user interface 700may include one or more user interface elements enabling a user ofclient computing device 130 to view raw test results data associatedwith the application testing controlled by software testing computingplatform 110, view test artifacts associated with the applicationtesting controlled by software testing computing platform 110, generatecustom reports associated with the application testing controlled bysoftware testing computing platform 110, and/or access other optionsassociated with the application testing controlled by software testingcomputing platform 110.

In some embodiments, receiving the request for the test resultsinformation from the client computing device may include receiving therequest for the test results information via a web dashboard provided tothe client computing device. For example, in receiving the request forthe test results information from the client computing device (e.g.,client computing device 130), software testing computing platform 110may receive the request for the test results information via a webdashboard provided to the client computing device (e.g., clientcomputing device 130) by software testing computing platform 110. Such aweb dashboard may, for instance, be provided by software testingcomputing platform 110 to client computing device 130 and/or may includeone or more of the example user interfaces discussed above.

In some embodiments, receiving the request for the test resultsinformation from the client computing device may include receiving therequest for the test results information via an application programminginterface. For example, in receiving the request for the test resultsinformation from the client computing device (e.g., client computingdevice 130), software testing computing platform 110 may receive therequest for the test results information via an application programminginterface. For instance, such an application programming interface mayexpose one or more functions provided by software testing computingplatform 110 to other applications on other devices, such as clientcomputing device 130, and an application executing on client computingdevice 130 may utilize one or more function calls provided by theapplication programming interface to request the test resultsinformation.

At step 211, software testing computing platform 110 may generate aresults report. For example, at step 211, in response to receiving therequest for the test results information from the client computingdevice (e.g., client computing device 130), software testing computingplatform 110 may generate a test results report. At step 212, softwaretesting computing platform 110 may send the results report to clientcomputing device 130. For example, at step 212, software testingcomputing platform 110 may send, via the communication interface (e.g.,communication interface 113), to the client computing device (e.g.,client computing device 130), the test results report.

FIGS. 3A, 3B, 3C, 3D, and 3E depict another illustrative event sequencefor conducting automated software testing using a centralized controllerand one or more distributed test host servers in accordance with one ormore example embodiments. The example event sequence depicted in FIGS.3A, 3B, 3C, 3D, and 3E may, for instance, illustrate a series of stepsexecuted in connection with the advanced architecture discussed above.

Referring to FIG. 3A, at step 301, software testing computing platform110 may receive test specification details information (e.g., fromenterprise server architecture 140 and/or from one or more othersources). For example, at step 301, prior to receiving a test executionrequest, software testing computing platform 110 may receive at least aportion of test specification details information from enterprise serverarchitecture (e.g., enterprise server architecture 140). The testspecification details information received by software testing computingplatform 110 from enterprise server architecture 140 may, for instance,include: information specifying one or more specific, different tests torun for one or more specific, different types of applications;information specifying one or more specific hosts and/or specific typesof hosts those tests should be run on; information specifying how tolaunch those tests (e.g., information specifying command line functions,arguments, parameters, and/or other launch information); and/or thelike. The tests that may be defined in the test specification detailsinformation received by software testing computing platform 110 fromenterprise server architecture 140 and/or subsequently run by softwaretesting computing platform 110 and/or associated testing architecturemay include any type of software application tests, such assecurity-related tests, server-side tests and/or service-basedarchitecture tests, fat-client and/or heavy-user-interface tests,user-application tests, web-based application tests, and/or other typesof tests.

At step 302, software testing computing platform 110 may update a testspecification database based on the test specification detailsinformation received from enterprise server architecture 140. Forexample, at step 302, software testing computing platform 110 may storethe at least the portion of the test specification details information(which may, e.g., be received from enterprise server architecture 140)in a test specification database (e.g., test specification database 112b).

At step 303, software testing computing platform 110 may receive a testexecution request (e.g., from administrative computing device 120,identifying a particular application to be tested and a particularenvironment for conducting such testing). For example, at step 303,software testing computing platform 110 may receive, via a communicationinterface (e.g., communication interface 113), from an administrativecomputing device (e.g., administrative computing device 120), a testexecution request. In some embodiments, receiving the test executionrequest from the administrative computing device may include receivinginformation identifying a specific software application to be tested ina specific computing environment. For example, in receiving the testexecution request from the administrative computing device (e.g.,administrative computing device 120), software testing computingplatform 110 may receive information identifying a specific softwareapplication to be tested in a specific computing environment. In someinstances, in receiving the test execution request from administrativecomputing device 120, software testing computing platform 110 mayreceive a message and/or other information from administrative computingdevice 120 corresponding to one or more selections made via one or moregraphical user interfaces presented by administrative computing device120, such as graphical user interface 400 depicted in FIG. 4, asdiscussed above.

At step 304, software testing computing platform 110 may load testspecification details information from a test specification databasebased on the test execution request. For example, at step 304, softwaretesting computing platform 110 may retrieve, from a test specificationdatabase (e.g., test specification database 112 b), test specificationdetails information based on the test execution request received fromthe administrative computing device (e.g., administrative computingdevice 120).

Referring to FIG. 3B, at step 305, software testing computing platform110 may identify one or more tests to execute based on the testspecification details information and the test execution request. Forexample, at step 305, software testing computing platform 110 mayidentify one or more tests to execute based on the test specificationdetails information retrieved from the test specification database(e.g., test specification database 112 b) and the test execution requestreceived from the administrative computing device (e.g., administrativecomputing device 120). Software testing computing platform 110 may, forinstance, identify the one or more tests to execute by matchinginformation from the test execution request (which may, e.g., specify aparticular application to be tested in a particular environment) withcorresponding test specification details information (which may, e.g.,specify what tests should be run to facilitate testing of the particularapplication, where such tests should be run, how such tests should belaunched, and/or the like) so as to initiate and/or complete testing ofthe specified application in the specified environment. In someinstances where the advanced architecture is utilized, a testing clientapplication executing on a test host (which may, e.g., be included inand/or otherwise associated with test host server farm 150) mayadditionally or alternatively identify the one or more tests to executeby matching information from the test execution request (which may,e.g., specify a particular application to be tested in a particularenvironment) with corresponding test specification details information(which may, e.g., specify what tests should be run to facilitate testingof the particular application, where such tests should be run, how suchtests should be launched, and/or the like).

At step 306, software testing computing platform 110 may generate one ormore local execution commands directing test host server farm 150 toexecute one or more tests (e.g., the one or more identified tests). Forexample, at step 306, software testing computing platform 110 maygenerate one or more local test execution commands directing a test hostserver farm (e.g., test host server farm 150) to execute the one or moretests identified based on the test specification details informationretrieved from the test specification database (e.g., test specificationdatabase 112 b) and the test execution request received from theadministrative computing device (e.g., administrative computing device120). In addition, the one or more local test execution commands (whichmay, e.g., be generated by software testing computing platform 110) maydirect the test host server farm (e.g., test host server farm 150) tolocally construct one or more command line instructions to be executedby the test host server farm (e.g., test host server farm 150) tocomplete the one or more tests. In this way, software testing computingplatform 110 may initiate software application testing at test hostserver farm 150 by sending one or more commands to test host server farm150, but test host server farm 150 may locally launch the appropriatetests by locally constructing and executing the command line testexecution commands.

In addition, in some instances, software testing computing platform 110may send, to test host server farm 150, information identifying the oneor more software application tests to be run on a particularapplication. For example, such information may be included in the one ormore commands generated and/or sent by software testing computingplatform 110 to test host server farm 150. In other instances, however,software testing computing platform 110 might not send informationidentifying the one or more software application tests to be run to testhost server farm 150. Rather, in these instances, test host server farm150 itself and/or a testing client application executed on test hostserver farm 150 may identify the one or more tests to run based on thecommands received from software testing computing platform 110 (whichmay, e.g., identify a particular application to be tested in aparticular environment) and test specification details informationretrieved by test host server farm 150 and/or the testing clientapplication executed on test host server farm 150 from the testspecification database (e.g., test specification database 112 b).

At step 307, software testing computing platform 110 may send the one ormore local execution commands to test host server farm 150. For example,at step 307, software testing computing platform 110 may send, via thecommunication interface (e.g., communication interface 113), to the testhost server farm (e.g., test host server farm 150), the one or morelocal test execution commands directing the test host server farm (e.g.,test host server farm 150) to execute the one or more tests identifiedbased on the test specification details information retrieved from thetest specification database (e.g., test specification database 112 b)and the test execution request received from the administrativecomputing device (e.g., administrative computing device 120).

At step 308, test host server farm 150 may receive the one or more localexecution commands from software testing computing platform 110. Forinstance, test host server farm 150 may receive the one or more localexecution commands from software testing computing platform 110 using alocal test client application executed on test host server farm 150 thatis configured to listen for and/or otherwise receive such commands.

Referring to FIG. 3C, at step 309, test host server farm 150 may locallylaunch one or more software application tests based on the one or morelocal execution commands received from software testing computingplatform 110. For example, at step 309, test host server farm 150 maylocally launch such software application tests using a local test clientapplication executed on test host server farm 150 that is configured tolocally launch, monitor, and/or otherwise control such testing on testhost server farm 150. Such a local test client application may, forinstance, implement and/or otherwise incorporate one or more aspects ofthe “TestAll Client” discussed above in connection with FIG. 1D.

At step 310, test host server farm 150 may locally capture test resultsdata. For example, at step 310, test host server farm 150 may locallycapture test results data using the local test client applicationexecuted on test host server farm 150 (which may, e.g., be configured tolocally capture raw test output and/or process such raw test output toproduce test results data). At step 311, test host server farm 150 maypost the test results data to a results service. For example, at step311, test host server farm 150 may post the test results data to aresults service provided by software testing computing platform 110using the local test client application executed on test host serverfarm 150 (which may, e.g., be configured to upload and/or otherwise sendthe test results data to software testing computing platform 110 and/ortest results database 112 e). At step 312, test host server farm 150 maypost test artifacts and/or other test output data to the resultsservice. For example, at step 312, test host server farm 150 may posttest artifacts and/or other test output data to the results serviceusing the local test client application executed on test host serverfarm 150 (which may, e.g., be configured to upload and/or otherwise sendthe test artifacts and/or the other test output data to software testingcomputing platform 110 and/or test artifact cloud 112 f).

Referring to FIG. 3D, at step 313, software testing computing platform110 may receive test results data (e.g., as a result of test host serverfarm 150 posting such test results data to a results service provided bysoftware testing computing platform 110). For example, at step 313,after sending the one or more local test execution commands to the testhost server farm (e.g., test host server farm 150), software testingcomputing platform 110 may receive, via a tests results service (whichmay, e.g., provided by test results service module 112 c), test resultsdata from the test host server farm (e.g., test host server farm 150).

In some embodiments, the one or more local test execution commands maycause a client application executed on the test host server farm to runone or more tests on the test host server farm that write pass-failoutput associated with the one or more tests to a console on the testhost server farm. For example, the one or more local test executioncommands (which may, e.g., be generated by software testing computingplatform 110 and sent by software testing computing platform 110 to testhost server farm 150) may cause a client application executed on thetest host server farm (e.g., test host server farm 150) to run one ormore tests on the test host server farm (e.g., test host server farm150) that write pass-fail output associated with the one or more teststo a console on the test host server farm (e.g., test host server farm150). As discussed above, the client application executed on the testhost server farm (e.g., test host server farm 150) may, for instance,implement and/or otherwise incorporate one or more aspects of the“TestAll Client” discussed above in connection with FIG. 1D.

In some embodiments, the one or more local test execution commands maycause the client application executed on the test host server farm tolocally capture output from the one or more tests run on the test hostserver farm. For example, the one or more local test execution commands(which may, e.g., be generated by software testing computing platform110 and sent by software testing computing platform 110 to test hostserver farm 150) may cause the client application executed on the testhost server farm (e.g., test host server farm 150) to locally captureoutput from the one or more tests run on the test host server farm(e.g., test host server farm 150).

In some embodiments, the one or more local test execution commands maycause the client application executed on the test host server farm tolocally process the output captured from the one or more tests run onthe test host server farm. For example, the one or more local testexecution commands (which may, e.g., be generated by software testingcomputing platform 110 and sent by software testing computing platform110 to test host server farm 150) may cause the client applicationexecuted on the test host server farm (e.g., test host server farm 150)to locally process the output captured from the one or more tests run onthe test host server farm (e.g., test host server farm 150).

In some embodiments, the one or more local test execution commands maycause the client application executed on the test host server farm tolocally process the output captured from the one or more tests run onthe test host server farm by determining, at the test host server farm,that a first set of tests passed based on the output captured from theone or more tests run on the test host server farm and by determining,at the test host server farm, that a second set of tests failed based onthe output captured from the one or more tests run on the test hostserver farm. For example, the one or more local test execution commands(which may, e.g., be generated by software testing computing platform110 and sent by software testing computing platform 110 to test hostserver farm 150) may cause the client application executed on the testhost server farm (e.g., test host server farm 150) to locally processthe output captured from the one or more tests run on the test hostserver farm (e.g., test host server farm 150) by determining, at thetest host server farm (e.g., test host server farm 150), that a firstset of tests passed based on the output captured from the one or moretests run on the test host server farm (e.g., test host server farm150). In addition, the one or more local test execution commands (whichmay, e.g., be generated by software testing computing platform 110 andsent by software testing computing platform 110 to test host server farm150) may cause the client application executed on the test host serverfarm (e.g., test host server farm 150) to locally process the outputcaptured from the one or more tests run on the test host server farm(e.g., test host server farm 150) by determining, at the test hostserver farm (e.g., test host server farm 150), that a second set oftests failed based on the output captured from the one or more tests runon the test host server farm (e.g., test host server farm 150).

At step 314, software testing computing platform 110 may update a testresults database (e.g., based on receiving test results data). Forexample, at step 314, software testing computing platform 110 may storethe test results data received from the test host server farm (e.g.,test host server farm 150) via the tests results service in a testresults database (e.g., test results database 112 e). In someembodiments, storing the test results data received from the test hostserver farm via the tests results service in the test results databasemay include storing timestamps indicating when one or more specifictests started and ended, information identifying a name of a specifichost on which the one or more specific tests were run, informationidentifying a number of assertions that passed, and informationidentifying a number of assertions that failed. For example, in storingthe test results data received from the test host server farm (e.g.,test host server farm 150) via the tests results service in the testresults database (e.g., test results database 112 e), software testingcomputing platform 110 may store timestamps indicating when one or morespecific tests started and ended, information identifying a name of aspecific host on which the one or more specific tests were run,information identifying a number of assertions that passed, informationidentifying a number of assertions that failed, and/or other informationassociated with the testing.

At step 315, software testing computing platform 110 may receive testoutput data (e.g., as a result of test host server farm 150 posting suchtest output data to a results service provided by software testingcomputing platform 110). For example, after sending the one or morelocal test execution commands to the test host server farm (e.g., testhost server farm 150), software testing computing platform 110 mayreceive, via the tests results service, test output data from the testhost server farm (e.g., test host server farm 150). In some instances,the test output data received by software testing computing platform 110from test host server farm 150 may include screenshots, raw test output,and/or other artifacts associated with the testing executed at test hostserver farm 150. At step 316, software testing computing platform 110may update the artifact cloud (e.g., based on receiving the test outputdata). For example, at step 316, software testing computing platform 110may store the test output data received from the test host server farm(e.g., test host server farm 150) via the tests results service in atest artifact cloud (e.g., test artifact cloud 112 f).

Referring to FIG. 3E, at step 317, software testing computing platform110 may receive a report request (e.g., from client computing device130). For example, at step 317, software testing computing platform 110may receive, via the communication interface (e.g., communicationinterface 113), from a client computing device (e.g., client computingdevice 130), a request for test results information. In some instances,in receiving the report request from client computing device 130,software testing computing platform 110 may receive a message and/orother information from client computing device 130 corresponding to oneor more selections made via one or more graphical user interfacespresented by client computing device 130, such as graphical userinterface 600, which is depicted in FIG. 6, and/or graphical userinterface 700, which is depicted in FIG. 7, as discussed above.

In some embodiments, receiving the request for the test resultsinformation from the client computing device may include receiving therequest for the test results information via a web dashboard provided tothe client computing device. For example, in receiving the request forthe test results information from the client computing device (e.g.,client computing device 130), software testing computing platform 110may receive the request for the test results information via a webdashboard provided to the client computing device (e.g., clientcomputing device 130) by software testing computing platform 110. Such aweb dashboard may, for instance, be provided by software testingcomputing platform 110 to client computing device 130 and/or may includeone or more of the example user interfaces discussed above.

In some embodiments, receiving the request for the test resultsinformation from the client computing device may include receiving therequest for the test results information via an application programminginterface. For example, in receiving the request for the test resultsinformation from the client computing device (e.g., client computingdevice 130), software testing computing platform 110 may receive therequest for the test results information via an application programminginterface. For instance, such an application programming interface mayexpose one or more functions provided by software testing computingplatform 110 to other applications on other devices, such as clientcomputing device 130, and an application executing on client computingdevice 130 may utilize one or more function calls provided by theapplication programming interface to request the test resultsinformation.

At step 318, software testing computing platform 110 may generate aresults report. For example, at step 318, in response to receiving therequest for the test results information from the client computingdevice (e.g., client computing device 130), software testing computingplatform 110 may generate a test results report. At step 319, softwaretesting computing platform 110 may send the results report to clientcomputing device 130. For example, at step 319, software testingcomputing platform 110 may send, via the communication interface (e.g.,communication interface 113), to the client computing device (e.g.,client computing device 130), the test results report.

The event sequences discussed above illustrate examples of how softwareapplication may be conducted using the basic architecture and advancedarchitecture described above in connection with FIGS. 1C and 1D. In someinstances, one or more steps of these example event sequences may beoptional and omitted accordingly. Additionally or alternatively, one ormore steps of these example event sequences may be performed in adifferent order than illustrated and/or may be repeated as variousdifferent applications are tested in different environments usingdifferent architectures.

FIG. 8 depicts an illustrative method for conducting automated softwaretesting using a centralized controller and one or more distributed testhost servers in accordance with one or more example embodiments.Referring to FIG. 8, at step 805, a computing platform having at leastone processor, a communication interface, and a memory may receive, viathe communication interface, from an administrative computing device, atest execution request. At step 810, the computing platform mayretrieve, from a test specification database, test specification detailsinformation based on the test execution request received from theadministrative computing device. At step 815, the computing platform mayidentify one or more tests to execute based on the test specificationdetails information retrieved from the test specification database andthe test execution request received from the administrative computingdevice. At step 820, the computing platform may generate one or moreremote test execution commands directing a test host server farm toexecute the one or more tests identified based on the test specificationdetails information retrieved from the test specification database andthe test execution request received from the administrative computingdevice. In addition, generating the one or more remote test executioncommands may include constructing one or more command line instructionsto be executed by the test host server farm and inserting the one ormore command line instructions to be executed by the test host serverfarm into the one or more remote test execution commands. At step 825,the computing platform may send, via the communication interface, to thetest host server farm, the one or more remote test execution commandsdirecting the test host server farm to execute the one or more testsidentified based on the test specification details information retrievedfrom the test specification database and the test execution requestreceived from the administrative computing device.

FIG. 9 depicts another illustrative method for conducting automatedsoftware testing using a centralized controller and one or moredistributed test host servers in accordance with one or more exampleembodiments. Referring to FIG. 9, at step 905, a computing platformhaving at least one processor, a communication interface, and a memorymay receive, via the communication interface, from an administrativecomputing device, a test execution request. At step 910, the computingplatform may retrieve, from a test specification database, testspecification details information based on the test execution requestreceived from the administrative computing device. At step 915, thecomputing platform may identify one or more tests to execute based onthe test specification details information retrieved from the testspecification database and the test execution request received from theadministrative computing device. At step 920, the computing platform maygenerate one or more local test execution commands directing a test hostserver farm to execute the one or more tests identified based on thetest specification details information retrieved from the testspecification database and the test execution request received from theadministrative computing device. In addition, the one or more local testexecution commands may direct the test host server farm to locallyconstruct one or more command line instructions to be executed by thetest host server farm to complete the one or more tests. At step 925,the computing platform may send, via the communication interface, to thetest host server farm, the one or more local test execution commandsdirecting the test host server farm to execute the one or more testsidentified based on the test specification details information retrievedfrom the test specification database and the test execution requestreceived from the administrative computing device.

One or more aspects of the disclosure may be embodied in computer-usabledata or computer-executable instructions, such as in one or more programmodules, executed by one or more computers or other devices to performthe operations described herein. Generally, program modules includeroutines, programs, objects, components, data structures, and the likethat perform particular tasks or implement particular abstract datatypes when executed by one or more processors in a computer or otherdata processing device. The computer-executable instructions may bestored as computer-readable instructions on a computer-readable mediumsuch as a hard disk, optical disk, removable storage media, solid-statememory, RAM, and the like. The functionality of the program modules maybe combined or distributed as desired in various embodiments. Inaddition, the functionality may be embodied in whole or in part infirmware or hardware equivalents, such as integrated circuits,application-specific integrated circuits (ASICs), field programmablegate arrays (FPGA), and the like. Particular data structures may be usedto more effectively implement one or more aspects of the disclosure, andsuch data structures are contemplated to be within the scope of computerexecutable instructions and computer-usable data described herein.

Various aspects described herein may be embodied as a method, anapparatus, or as one or more computer-readable media storingcomputer-executable instructions. Accordingly, those aspects may takethe form of an entirely hardware embodiment, an entirely softwareembodiment, an entirely firmware embodiment, or an embodiment combiningsoftware, hardware, and firmware aspects in any combination. Inaddition, various signals representing data or events as describedherein may be transferred between a source and a destination in the formof light or electromagnetic waves traveling through signal-conductingmedia such as metal wires, optical fibers, or wireless transmissionmedia (e.g., air or space). In general, the one or morecomputer-readable media may be and/or include one or more non-transitorycomputer-readable media.

As described herein, the various methods and acts may be operativeacross one or more computing servers and one or more networks. Thefunctionality may be distributed in any manner, or may be located in asingle computing device (e.g., a server, a client computer, and thelike). For example, in alternative embodiments, one or more of thecomputing platforms discussed above may be combined into a singlecomputing platform, and the various functions of each computing platformmay be performed by the single computing platform. In such arrangements,any and/or all of the above-discussed communications between computingplatforms may correspond to data being accessed, moved, modified,updated, and/or otherwise used by the single computing platform.Additionally or alternatively, one or more of the computing platformsdiscussed above may be implemented in one or more virtual machines thatare provided by one or more physical computing devices. In sucharrangements, the various functions of each computing platform may beperformed by the one or more virtual machines, and any and/or all of theabove-discussed communications between computing platforms maycorrespond to data being accessed, moved, modified, updated, and/orotherwise used by the one or more virtual machines.

Aspects of the disclosure have been described in terms of illustrativeembodiments thereof. Numerous other embodiments, modifications, andvariations within the scope and spirit of the appended claims will occurto persons of ordinary skill in the art from a review of thisdisclosure. For example, one or more of the steps depicted in theillustrative figures may be performed in other than the recited order,and one or more depicted steps may be optional in accordance withaspects of the disclosure.

What is claimed is:
 1. A computing platform, comprising: at least oneprocessor; a communication interface communicatively coupled to the atleast one processor; and memory storing computer-readable instructionsthat, when executed by the at least one processor, cause the computingplatform to: receive, via the communication interface, from anadministrative computing device, a test execution request; retrieve,from a test specification database, test specification detailsinformation based on the test execution request received from theadministrative computing device; identify one or more tests to executebased on the test specification details information retrieved from thetest specification database and the test execution request received fromthe administrative computing device; generate one or more local testexecution commands directing a test host server farm to execute the oneor more tests identified based on the test specification detailsinformation retrieved from the test specification database and the testexecution request received from the administrative computing device,wherein the one or more local test execution commands direct the testhost server farm to locally construct one or more command lineinstructions to be executed by the test host server farm to complete theone or more tests; and send, via the communication interface, to thetest host server farm, the one or more local test execution commandsdirecting the test host server farm to execute the one or more testsidentified based on the test specification details information retrievedfrom the test specification database and the test execution requestreceived from the administrative computing device.
 2. The computingplatform of claim 1, wherein receiving the test execution request fromthe administrative computing device comprises receiving informationidentifying a specific software application to be tested in a specificcomputing environment.
 3. The computing platform of claim 1, wherein thememory stores additional computer-readable instructions that, whenexecuted by the at least one processor, cause the computing platform to:after sending the one or more local test execution commands to the testhost server farm, receive, via a tests results service, test resultsdata from the test host server farm; and store the test results datareceived from the test host server farm via the tests results service ina test results database.
 4. The computing platform of claim 3, whereinthe one or more local test execution commands cause a client applicationexecuted on the test host server farm to run one or more tests on thetest host server farm that write pass-fail output associated with theone or more tests to a console on the test host server farm.
 5. Thecomputing platform of claim 4, wherein the one or more local testexecution commands cause the client application executed on the testhost server farm to locally capture output from the one or more testsrun on the test host server farm.
 6. The computing platform of claim 5,wherein the one or more local test execution commands cause the clientapplication executed on the test host server farm to locally process theoutput captured from the one or more tests run on the test host serverfarm.
 7. The computing platform of claim 5, wherein the one or morelocal test execution commands cause the client application executed onthe test host server farm to locally process the output captured fromthe one or more tests run on the test host server farm by determining,at the test host server farm, that a first set of tests passed based onthe output captured from the one or more tests run on the test hostserver farm and by determining, at the test host server farm, that asecond set of tests failed based on the output captured from the one ormore tests run on the test host server farm.
 8. The computing platformof claim 3, wherein storing the test results data received from the testhost server farm via the tests results service in the test resultsdatabase comprises storing timestamps indicating when one or morespecific tests started and ended, information identifying a name of aspecific host on which the one or more specific tests were run,information identifying a number of assertions that passed, andinformation identifying a number of assertions that failed.
 9. Thecomputing platform of claim 3, wherein the memory stores additionalcomputer-readable instructions that, when executed by the at least oneprocessor, cause the computing platform to: after sending the one ormore local test execution commands to the test host server farm,receive, via the tests results service, test output data from the testhost server farm; and store the test output data received from the testhost server farm via the tests results service in a test artifact cloud.10. The computing platform of claim 1, wherein the memory storesadditional computer-readable instructions that, when executed by the atleast one processor, cause the computing platform to: prior to receivingthe test execution request from the administrative computing device:receive at least a portion of the test specification details informationfrom enterprise server architecture; and store the at least the portionof the test specification details information in the test specificationdatabase.
 11. The computing platform of claim 1, wherein the memorystores additional computer-readable instructions that, when executed bythe at least one processor, cause the computing platform to: receive,via the communication interface, from a client computing device, arequest for test results information; in response to receiving therequest for the test results information from the client computingdevice, generate a test results report; and send, via the communicationinterface, to the client computing device, the test results report. 12.The computing platform of claim 11, wherein receiving the request forthe test results information from the client computing device comprisesreceiving the request for the test results information via a webdashboard provided to the client computing device.
 13. The computingplatform of claim 11, wherein receiving the request for the test resultsinformation from the client computing device comprises receiving therequest for the test results information via an application programminginterface.
 14. The computing platform of claim 1, wherein sending theone or more local test execution commands to the test host server farmcauses the test host server farm to locally launch the one or more testsat the test host server farm by locally constructing and executing theone or more command line instructions at the test host server farm. 15.A method, comprising: at a computing platform comprising at least oneprocessor, memory, and a communication interface: receiving, by the atleast one processor, via the communication interface, from anadministrative computing device, a test execution request; retrieving,by the at least one processor, from a test specification database, testspecification details information based on the test execution requestreceived from the administrative computing device; identifying, by theat least one processor, one or more tests to execute based on the testspecification details information retrieved from the test specificationdatabase and the test execution request received from the administrativecomputing device; generating, by the at least one processor, one or morelocal test execution commands directing a test host server farm toexecute the one or more tests identified based on the test specificationdetails information retrieved from the test specification database andthe test execution request received from the administrative computingdevice, wherein the one or more local test execution commands direct thetest host server farm to locally construct one or more command lineinstructions to be executed by the test host server farm to complete theone or more tests; and sending, by the at least one processor, via thecommunication interface, to the test host server farm, the one or morelocal test execution commands directing the test host server farm toexecute the one or more tests identified based on the test specificationdetails information retrieved from the test specification database andthe test execution request received from the administrative computingdevice.
 16. The method of claim 15, wherein receiving the test executionrequest from the administrative computing device comprises receivinginformation identifying a specific software application to be tested ina specific computing environment.
 17. The method of claim 15,comprising: after sending the one or more local test execution commandsto the test host server farm, receiving, by the at least one processor,via a tests results service, test results data from the test host serverfarm; and storing, by the at least one processor, the test results datareceived from the test host server farm via the tests results service ina test results database.
 18. The method of claim 17, wherein the one ormore local test execution commands cause a client application executedon the test host server farm to run one or more tests on the test hostserver farm that write pass-fail output associated with the one or moretests to a console on the test host server farm.
 19. The method of claim18, wherein the one or more local test execution commands cause theclient application executed on the test host server farm to locallycapture output from the one or more tests run on the test host serverfarm.
 20. One or more non-transitory computer-readable media storinginstructions that, when executed by a computing platform comprising atleast one processor, memory, and a communication interface, cause thecomputing platform to: receive, via the communication interface, from anadministrative computing device, a test execution request; retrieve,from a test specification database, test specification detailsinformation based on the test execution request received from theadministrative computing device; identify one or more tests to executebased on the test specification details information retrieved from thetest specification database and the test execution request received fromthe administrative computing device; generate one or more local testexecution commands directing a test host server farm to execute the oneor more tests identified based on the test specification detailsinformation retrieved from the test specification database and the testexecution request received from the administrative computing device,wherein the one or more local test execution commands direct the testhost server farm to locally construct one or more command lineinstructions to be executed by the test host server farm to complete theone or more tests; and send, via the communication interface, to thetest host server farm, the one or more local test execution commandsdirecting the test host server farm to execute the one or more testsidentified based on the test specification details information retrievedfrom the test specification database and the test execution requestreceived from the administrative computing device.